Tolerating bad health research: the continuing scandal.

BACKGROUND: At the 2015 REWARD/EQUATOR conference on research waste, the late Doug Altman revealed that his only regret about his 1994 BMJ paper 'The scandal of poor medical research' was that he used the word 'poor' rather than 'bad'. But how much research is bad? And what would improve things? MAIN TEXT: We focus on randomised trials and look at scale, participants and cost. We randomly selected up to two quantitative intervention reviews published by all clinical Cochrane Review Groups between May 2020 and April 2021. Data including the risk of bias, number of participants, intervention type and country were extracted for all trials included in selected reviews. High risk of bias trials was classed as bad. The cost of high risk of bias trials was estimated using published estimates of trial cost per participant. We identified 96 reviews authored by 546 reviewers from 49 clinical Cochrane Review Groups that included 1659 trials done in 84 countries. Of the 1640 trials providing risk of bias information, 1013 (62%) were high risk of bias (bad), 494 (30%) unclear and 133 (8%) low risk of bias. Bad trials were spread across all clinical areas and all countries. Well over 220,000 participants (or 56% of all participants) were in bad trials. The low estimate of the cost of bad trials was £726 million; our high estimate was over £8 billion. We have five recommendations: trials should be neither funded (1) nor given ethical approval (2) unless they have a statistician and methodologist; trialists should use a risk of bias tool at design (3); more statisticians and methodologists should be trained and supported (4); there should be more funding into applied methodology research and infrastructure (5). CONCLUSIONS: Most randomised trials are bad and most trial participants will be in one. The research community has tolerated this for decades. This has to stop: we need to put rigour and methodology where it belongs - at the centre of our science.

A perspective on new dilemmas in scientific publishing in the time of a pandemic.

BACKGROUND: The COVID-19 pandemic exerted manifold pressures on the public health framework globally, but it also in a way unified different genres and allowed for strategizing and implementing regulatory decisions as best as possible, especially in India. There is an unmet need for such a unified and integrative approach in the area of scientific publishing which has also been touched by various dilemmas, either emergent or propagated during this pandemic. OBJECTIVES: This article intends to re-visit some of the dilemmas in scientific publishing, which have taken centre stage owing to a healthcare emergency, with the objective of highlighting an unmet need for developing unified criteria for research conduction and publishing from a futuristic view point, as one is not without the other. CONTENT: While a fast track delivery of research data has been a priority for research journals, the due pressures in the process management of the same while skimming the ethical boundaries of responsible mediation through a Journal platform has remained a challenge globally for various reasons. Furthermore, the inevitability of a healthcare emergency inadvertently led to some cumulative off-target effects including accumulation of research waste, diminishing validity of academic metrics, short data set publications, hasty zombie clinical trials publishing merely an overview of the actual data, etc, which are major issues not only for journal Editors or the research community as a whole, but also for regulatory authorities and policy makers. As a step towards future pandemic preparedness, strategizing and streamlining research and publication processes ensuing responsible reporting should be treated as a topic of paramount significance. Hence, through debating on these dilemmas as well as potential integrative approaches, unified guiding criteria in the area of scientific publishing may be developed in lieu of preparedness for such future pandemic scenarios.

Review finds core outcome set uptake in new studies and systematic reviews needs improvement.

OBJECTIVES: To review evidence about the uptake of core outcome sets (COS). A COS is an agreed standardized set of outcomes that should be measured and reported, as a minimum, in all clinical trials in a specific area of health or healthcare. STUDY DESIGN AND SETTING: This article provides an analysis of what is known about the uptake of COS in research. Similarities between COS and outcomes recommended by stakeholders in the evidence ecosystem is reviewed and actions taken by them to facilitate COS uptake described. RESULTS: COS uptake is low in most research areas. Common facilitators relate to trialist awareness and understanding. Common barriers were not including in the development process all specialties that might use the COS and the lack of recommendations for how to measure the outcomes. Increasingly, COS developers are considering strategies for promoting uptake earlier in the process, including actions beyond traditional dissemination approaches. An overlap between COS and outcomes in regulatory documents and health technology assessments is good. An increasing number and variety of organizations are recommending COS be considered. CONCLUSION: We suggest actions for various stakeholders for improving COS uptake. Research is needed to assess the impact of these actions to identify effective evidence-based strategies.

Systematic reviews of convalescent plasma in COVID-19 continue to be poorly conducted and reported: a systematic review.

OBJECTIVES: To suggest possible approaches to combatting the impact of the COVID-19 infodemic to prevent research waste in future health emergencies and in everyday research and practice. STUDY DESIGN AND SETTING: Systematic review. The Epistemonikos database was searched in June 2021 for systematic reviews on the effectiveness of convalescent plasma for COVID-19. Two reviewers independently screened the retrieved references with disagreements resolved by discussion. Data extraction was completed by one reviewer with a proportion checked by a second. We used the Assessment of Multiple Systematic Reviews to assess the quality of conduct and reporting of included reviews. RESULTS: Fifty one systematic reviews are included with 193 individual studies included within the systematic reviews. There was considerable duplication of effort; multiple reviews were conducted at the same time with inconsistencies in the evidence included. The reviews were of low methodological quality, poorly reported, and did not adhere to preferred reporting items for systematic reviews and meta-analysis guidance. CONCLUSION: Researchers need to conduct, appraise, interpret, and disseminate systematic reviews better. All in the research community (researchers, peer-reviewers, journal editors, funders, decision makers, clinicians, journalists, and the public) need to work together to facilitate the conduct of robust systematic reviews that are published and communicated in a timely manner, reducing research duplication and waste, increasing transparency and accessibility of all systematic reviews.

Saving millions of lives but some resources squandered: emerging lessons from health research system pandemic achievements and challenges.

During the SARS-CoV-2 pandemic, astonishingly rapid research averted millions of deaths worldwide through new vaccines and repurposed and new drugs. Evidence use informed life-saving national policies including non-pharmaceutical interventions. Simultaneously, there was unprecedented waste, with many underpowered trials on the same drugs. We identified lessons from COVID-19 research responses by applying WHO's framework for research systems. It has four functions-governance, securing finance, capacity-building, and production and use of research-and nine components. Two linked questions focused the analysis. First, to what extent have achievements in knowledge production and evidence use built on existing structures and capacity in national health research systems? Second, did the features of such systems mitigate waste? We collated evidence on seven countries, Australia, Brazil, Canada, Germany, New Zealand, the United Kingdom and the United States, to identify examples of achievements and challenges.We used the data to develop lessons for each framework component. Research coordination, prioritization and expedited ethics approval contributed to rapid identification of new therapies, including dexamethasone in the United Kingdom and Brazil. Accelerated vaccines depended on extensive funding, especially through the Operation Warp Speed initiative in the United States, and new platforms created through long-term biomedical research capacity in the United Kingdom and, for messenger ribonucleic acid (mRNA) vaccines, in Canada, Germany and the United States. Research capacity embedded in the United Kingdom's healthcare system resulted in trial acceleration and waste avoidance. Faster publication of research saved lives, but raised challenges. Public/private collaborations made major contributions to vastly accelerating new products, available worldwide, though unequally. Effective developments of living (i.e. regularly updated) reviews and guidelines, especially in Australia and Canada, extended existing expertise in meeting users' needs. Despite complexities, effective national policy responses (less evident in Brazil, the United Kingdom and the United States) also saved lives by drawing on health research system features, including collaboration among politicians, civil servants and researchers; good communications; and willingness to use evidence. Comprehensive health research strategies contributed to success in research production in the United Kingdom and in evidence use by political leadership in New Zealand. In addition to waste, challenges included equity issues, public involvement and non-COVID research. We developed recommendations, but advocate studies of further countries.

We need to talk about solutions looking through the lens of evidence syntheses of convalescent plasma therapy: a systematic review

BackgroundThe COVID-19 pandemic resulted in a substantial amount of research produced with startling speed. While the need to understand the nature of the SARS-COV-2 virus, its spread, impact and possible treatments quickly was necessary, we have observed a plethora of variable quality systematic reviews undermining the confidence associated with these methods and challenging the use of evidence to inform practice. To better understand the issues underlying the infodemic of COVID-19 related evidence, we have used an exemplar topic – the effectiveness of convalescent plasma therapy for COVID-19 – to explore the timelines, characteristics and methods used in this body of evidence and offer solutions to avoid similar research waste in future health emergencies.MethodsThe Epistemonikos database was searched on 8th June 2021 for systematic reviews on the effectiveness of convalescent plasma for COVID-19 using the phrase ‘convalescent plasma’ in the COVID-19 evidence section. Any, quantitative, systematic review related to the effectiveness or safety of convalescent plasma in treating COVID-19in hospitalised patients was included. Study selection, data extraction and quality assessment (using AMSTAR-2) were conducted and checked by a team of reviewers. Data were tabulated and explored using interactive visualisation methods including network analysis. Issues in the conduct and reporting of systematic reviews and potential solutions were established by consensus with supporting evidence identified from the literature.Results51 systematic reviews were included from 24 countries. We found considerable duplication of effort with 48 reviews on the same topic published within the 17 month time frame of the study. There were inconsistencies in the evidence included within the reviews;of a total of 193 individual primary studies, 75% were included in three or fewer reviews. The reviews were of low methodological quality (45 assessed as critically low on AMSTAR-2) and many did not adhere to PRISMA reporting guidance.ConclusionAs researchers we need to conduct, appraise, interpret and disseminate systematic reviews better. To prevent repetition of what we have observed in the COVID-19 infodemic, we need to promote a greater understanding that not all evidence is equal, value robust methods (even if they take more time), consider what the research adds before we conduct it or replicate it) and keep all communication clear (easy to understand and accessible). We welcome further thoughts and discussions on new ways to resolve these issues to help the research community move forward with positive, dynamic and agile strategies.

Tracing open data in emergencies: The case of the COVID-19 pandemic.

BACKGROUND: The coronavirus disease 2019 (COVID-19) pandemic constitutes an ongoing, burning Public Health Emergency of International Concern (PHEIC). In 2015, the World Health Organization adopted an open data policy recommendation in such situations. OBJECTIVES: The present cross-sectional meta-research study aimed to assess the availability of open data and metrics of articles pertaining to the COVID-19 outbreak in five high-impact journals. METHODS: All articles regarding the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), published in five high-impact journals (Ann Intern Med, BMJ, JAMA, NEJM and Lancet) until March 14, 2020 were retrieved. Metadata (namely the type of article, number of authors, number of patients, citations, errata, news and social media mentions) were extracted for each article in each journal in a systematic way. Google Scholar and Scopus were used for citations and author metrics respectively, and Altmetrics and PlumX were used for news and social media mentions retrieval. The degree of adherence to the PHEIC open data call was also evaluated. RESULTS: A total of 140 articles were published until March 14, 2020, mostly opinion papers. Sixteen errata followed these publications. The number of authors in each article ranged from 1 to 63, whereas the number of patients with a laboratory-confirmed SARS-CoV-2 infection reached 2645. Extensive hyperauthorship was evident among case studies. The impact of these publications reached a total of 4210 cumulative crude citations and 342 790 news and social media mentions. Only one publication (0.7%) provided complete open data, while 32 (22.9%) included patient data. CONCLUSIONS: Even though a large number of manuscripts was produced since the pandemic, availability of open data remains restricted.